nameparser 2.0 implementation#288
Draft
derek73 wants to merge 218 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v2 core design uses enum.StrEnum (3.11+). Full #257 (dropping typing_extensions, CI matrix) remains tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Single letter, bare or period-written; of the default suffix words that means exactly the roman numerals I and V. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Yamamoto, Haruki' only demonstrates the point if you already know which word is the surname. 'Thomas, John' is ambiguous to an English-reading audience in exactly the way the comma resolves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Append ellipses to the field docs' example parentheticals and teach the inspection idiom once in the class docstring (Lexicon.default().<field> is the full list). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each field doc now ends with a same-page link to the config data constant it sources from (the constants render their complete contents in the reference's compat section). The renamed pairs (suffix_words -> SUFFIX_NOT_ACRONYMS, bound_given_names -> BOUND_FIRST_NAMES) double as visible v1<->v2 mapping documentation, and particles_ambiguous -- which has no constant -- documents its derivation from NON_FIRST_NAME_PREFIXES instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same enum-member invisibility as AmbiguityKind: without doc comments autodoc skipped them, so the class docstring promised seven fields the page never listed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fold example (verified live): 'Gabriel García Márquez' -> family 'García Márquez' for multi-part-surname data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review feedback (Derek): 'How the parser thinks' implied nondeterminism -> 'How the parser works'; the 'four short essays' opener and 'pipeline, in one breath' phrasing replaced with direct statements; 'From string to name' now walks the canonical example concretely (8 tokens, Dr. span (0,3), family as a view joining de/la/Vega) and defines span before explaining what it prevents; the three-container bullets lead with the container names; the Dean paragraph returns to the v1 docs' problem-first framing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DEFAULT_NICKNAME_DELIMITERS grows by the eight conventions the issue
proposed: smart quotes, low-high (de/pl/cs/hu), right-right (sv/fi),
guillemets both directions (« Petit » inner spacing tolerated), CJK
corner brackets, fullwidth parentheses. Curly single quotes stay
excluded (U+2019 is the apostrophe in O'Connor, pinned). Both APIs
get them: the shim's sentinel map gains named keys
(smart_double_quotes, guillemets, ...) so the v1 keyed idioms
(pop/move/del) work on the new pairs like the original trio.
The conventions share characters in opposite roles ('"' closes the
German pair but opens the English one; '»' closes guillemets and
opens the reversed pair), which made extraction emit spurious
unbalanced-delimiter ambiguities: extract now tracks unmatched-open
offsets and drops candidates inside a region another pair
successfully masked. Genuine unbalanced opens still flag.
TDD throughout (extract suppression -> defaults -> shim sentinels,
each red first); 9 case rows through both runners; differential
harness exit 0 unchanged (no typographic quotes in the corpus);
release-log feat entry; reference/customize/migrate docs updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BOUND_FIRST_NAMES had only the transliterations; the script originals now join too: عبد (one entry covers abdul/abdel/abdal -- script attaches the article to the following word), the kunya pair أبو/ابو, and أم/ام. Behavior mirrors the Latin twins, probed live before pinning: bound join fires with 3+ tokens (عبد الرحمن محمد -> given 'عبد الرحمن'), the two-token kunya stays split (أبو مازن unchanged), and non-leading أبو still prefix-chains onto family (أحمد أبو خليل unchanged -- both prior pins hold). TDD: 4 case rows red in both runners first. Differential exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Twelve given-name titles -- Arabic honorifics precede the given name,
so all join الشيخ in FIRST_NAME_TITLES: the article forms (never
given names) الدكتور/الدكتورة/الأستاذ/الأستاذة/الحاج/الحاجة/الشيخة
and the bare doctor/professor/engineer forms
دكتور/دكتورة/أستاذ/أستاذة/مهندس. Deferred under the collision rule
(recorded in the data-module comment): bare سيد/شيخ/أمير/سلطان (all
common given names), the د. abbreviation (bare د would swallow
single-letter initials, the bare-κ trap), and the Ottoman
post-nominals باشا/بك/أفندي (survive as family names).
و ('and') joins conjunctions: formal script attaches it to the next
word, but the standalone informal spacing is common in real data;
single-character like y/и, so the #11 carve-out protects short names.
TDD: 20 rows red first, incl. the الحاج + عبد الرحمن compound
(title + bound join + family) and the 6-piece و title chain.
Differential exit 0 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ow-up) Hebrew titles גברת/פרופ'/פרופ׳/פרופסור/עו"ד/עו״ד/הרב join מר as plain titles (Israeli convention: family follows), each abbreviation in both geresh/gershayim spellings per the ד"ר precedent. Hebrew gains the suffix category Arabic deferred: ז"ל and שליט"א (both spellings) -- post-nominals ubiquitous in genealogical data, never names, mid-word quotes inert in extraction. Devanagari becomes a #269 script: श्री/श्रीमती/डॉ (डॉ. matches via edge-period normalization). Latin sri/shri deliberately NOT added -- the transliterations collide with real given names (Sri Mulyani); the native script cannot. Deferred under the collision rule, recorded in the data-module comments: bare רב (the ordinary word 'many') and בר as a particle (Bar is a common modern Israeli given name; the surname spelling is hyphenated anyway). TDD: 15 rows red first across title/suffix/script categories. Differential exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three review agents independently found the same defect in the #273 defaults: each pair scanned the full text independently in sorted order, so when two conventions sharing a character in opposite roles BOTH occurred ('Hans „Erster" und "Zweiter" Müller'), the earlier-sorted pair claimed the other's close as its own open, extracted a bogus span across the real boundary, and the legitimate match was dropped by the overlap check with zero ambiguity signal. extract_delimited now runs one position-driven scan: at each position the leftmost boundary-valid opener among ALL pairs wins, with per-pair forward-only cursors keeping the scan linear. Overlapping matches become impossible by construction, so the overlap check on matches is gone; the unmatched-open filter remains (now via _overlaps) for bulk-recorded dangling opens consumed by later matches. Bucket precedence is documented where it actually lives: Policy's maiden-wins canonicalization (facade: shim pre-subtraction) -- the stale 'maiden scanned first' module-docstring claim and the contradicting 'nickname first' comment are both gone. Also the review's quantified perf fix: a cached delimiter-charset prescreen short-circuits the no-delimiter common case (isdisjoint), more than paying back the 3->11 default-pair growth. TDD: four mixed-convention/coexistence tests red first; 19 extract tests green; differential exit 0 unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- name_order's comma claim narrowed: only a family-separating comma ignores it; 'John Smith, Jr.' still obeys name_order (live-proven, SUFFIX_COMMA reads it). - The 'issue #11' carve-out citations were pointing at GitHub #11 (a 2014 'meant to create a milestone' accident); the real source is GOOGLE CODE issue 11, the 'john e smith' bug -- v1's own comment carries the full URL. All six sites now say so. - _DelimiterManager no longer claims 'only the three named sentinels'. - The בר particle deferral moved to prefixes.py with the other particle decisions. - The patronymic migration hint is a single module constant shared by Policy and PolicyPatch. - apply_patch documents maiden-wins through patches as intended (decided 2026-07-19): maiden membership IS the routing decision, whoever contributes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Match-time contract of lower()-not-casefold() pinned at parse level (ΚΟΣ and GROẞFÜRST match; ASCII-SS GROSSFÜRST deliberately does not, v1 parity); facade rows for each new vocabulary category (bound given, Hebrew suffix, Devanagari title); maiden-wins pinned on a typographic pair; a pre-#273 three-key pickle restores as exactly its own keys (setstate replaces, never merges defaults). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"core spec §5a/§5b/§6" pointed at docs/superpowers/, which is gitignored and not committed on any branch -- a future reader can never resolve the citation. Replace each with the actual rule content inline instead (review finding on #288).
patronymic_rules already probed with iter() to give non-iterable values (an int, a bool) a curated, field-naming TypeError. The other iterable-typed fields (name_order, nickname_delimiters, maiden_delimiters, extra_suffix_delimiters, every Lexicon vocab field, capitalization_exceptions) fell through to whatever bare "'int' object is not iterable" the first tuple()/iter() call happened to raise. Apply the same _require_iterable probe everywhere, with regression tests (review finding on #288).
role = Role.GIVEN when a main-stream token reaches assemble() with no role set was previously untested -- assign/group always set a role in the real pipeline, so nothing exercised this last-resort branch. Construct the invariant violation directly (an unassigned-role WorkToken) so a future refactor can't silently change or lose this documented safety-net behavior (review finding on #288).
The layering rule forbids the pipeline/_render from importing nameparser.config directly, so several patterns are copied by hand with "keep in sync by hand" comments -- nothing previously checked that promise. Tests may legally import both sides (test_layering.py's own convention), so add sync tests for every duplicated pattern/table, including the documented "initial" trailing-"?" deviation between config and the pipeline copies (review finding on #288).
Several branches were only reached through tests/v2/test_cases.py's full-pipeline corpus, so a regression scoped to one stage wouldn't be pinpointed by that stage's own test file: assign's SUFFIX_COMMA structure, segment's cosmetic-vs-structural empty-bucket handling (single trailing comma vs. a leading comma), group's extra_suffix_delimiters entry-splitting in a FAMILY_COMMA tail segment, and tokenize's interleaving of two simultaneously active extracted regions (review finding on #288).
"always a subset" read as identity to a casual reader, but the constructor check uses `in` (Token's == equality): two distinct Token instances with identical text/span/role/tags satisfy it. Document the distinction at the field, the class docstring, and the check site (polish item from the #288 review).
name_order and the union (set-valued) fields get an iter()/type probe in __post_init__; the four scalar override fields (middle_as_family, lenient_comma_suffixes, strip_emoji, strip_bidi) deliberately get none until apply_patch runs -- previously true only per the class docstring and the tests, with nothing at the loop itself flagging the omission as intentional rather than a gap (polish item from the #288 review).
The prose in each comment already names the specific v1 logic being ported; the hardcoded line numbers (1285, 1313, 1333, 1368, 1390) add nothing on top of that and would silently go stale on the next edit to parser.py, with nothing to catch the drift (polish item from the #288 review).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Retitle "three containers" section and swap the Policy example from name_order (which actually correlates with language for Chinese/ Hungarian/Vietnamese names) to nickname_delimiters, which doesn't. - Note that capitalization_exceptions keys match punctuation-normalized text, so "phd" also covers "Ph.D.". - Fix a stale nickname_delimiters example (guillemets are already in the default set) and drop an unneeded frozenset() wrapper. - Replace the rendering-arguments stub signatures with links to the real methods plus a worked, doctested example, including capitalized(force=True) for genuinely mixed-case input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Derek's terminal find: str(parse('Von Johnson (smith)')) silently
dropped the nickname. v1's default string_format always rendered
'({nickname})'; the 2.0 default had lost it. The default spec is now
'{title} {given} {middle} {family} {suffix} ({nickname}) née
{maiden}' -- v1's nickname position restored, maiden appended with a
marker that round-trips (parens re-extract as nickname, née is a
maiden marker, pinned by a reparse assertion). The #254 collapse
learns to drop the trailing orphaned 'née' (and the bare-née
all-empty case); content 'Née' is untouched (case-exact match). The
HumanName facade keeps v1's own string_format default.
TDD: red first; differential exit 0 (facade rendering unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hesized maiden
'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}':
nickname in the classic quoted position after the given name, maiden
parenthesized after the family. The collapse's existing empty-'""'
and empty-'()' rules cover both decorations, so the née trailing-strip
special case is deleted. Trade-off documented: the quoted nickname
round-trips exactly; parenthesized maiden re-parses as a nickname
(custom 'née {maiden}' spec available for lossless round-trip).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness) — full v1 suite (1,240+ tests) reconciled and passing against the facade; differential harness (tools/differential/) verifies 1.4-on-PyPI vs the facade over 486 corpus names with one classified diffru,tr_az) — shipped with the non-interference gate, Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269 default vocabulary, and CLI--localeNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code